feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult - #732
Conversation
|
NB: I'm still investigating what led to such a large diff in the protobuf generated file. The changes correspond only to compiling the PR at apache/arrow#49498 with the following command: @zeroshade am I supposed to merge changes to that file? I left them here for now to make the PR self contained for easier review, but I assume that if the PR in the main repo is merged, the generated code in this repo will be automatically updated. I will remove the changes if/when that time comes. |
The diff is likely due to the fact that we haven't regenerated the protobuf code for quite a while, and protoc is now several versions ahead of when we last regenerated the code. We also have a go generate that'll regenerate for you btw. I don't consider the diff in the protobuf to necessarily be bad though |
b8d7095 to
a432344
Compare
|
I rebased this onto main which should solve the CI issues, I'll also re-review it |
There was a problem hiding this comment.
Pull request overview
This PR extends the FlightSQL prepared statement API to surface a server-provided optional is_update hint (as *bool) so clients can choose the correct execution path (query vs update) when using prepared statements.
Changes:
- Added
IsUpdate *boolto the server-sideActionCreatePreparedStatementResultand mapped it into the protobuf action result. - Plumbed
IsUpdatethrough the client prepared statement parsing/loading path and exposed it viaPreparedStatement.IsUpdate(). - Added unit tests (client) and integration tests (server) covering both
is_update=false(query) andis_update=true(update) prepared statements.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
arrow/flight/flightsql/server.go |
Adds IsUpdate to the server result struct and forwards it for CreatePreparedStatement responses (but currently not for Substrait plan responses). |
arrow/flight/flightsql/client.go |
Stores/parses IsUpdate into PreparedStatement and exposes a getter; updates related documentation. |
arrow/flight/flightsql/client_test.go |
Adds unit tests verifying the client reads IsUpdate and executes the appropriate code path. |
arrow/flight/flightsql/server_test.go |
Adds integration tests (and server stubs) to validate end-to-end behavior for prepared statement query/update execution and the IsUpdate hint. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| query := req.GetQuery() | ||
| var isUpdate *bool | ||
| result.Handle = []byte(query) | ||
| switch query { | ||
| case "prepared query": | ||
| isUpdate := false | ||
| result.IsUpdate = &isUpdate | ||
| case "prepared update": | ||
| isUpdate := true | ||
| result.IsUpdate = &isUpdate | ||
| default: | ||
| err = fmt.Errorf("unknown query: %s", query) | ||
| } | ||
| if isUpdate != nil { | ||
| result.IsUpdate = isUpdate | ||
| } | ||
| return |
There was a problem hiding this comment.
Good catch. This was also highlighted by Matt and I addressed it in #732 (comment)
| if output.ParameterSchema != nil { | ||
| result.ParameterSchema = flight.SerializeSchema(output.ParameterSchema, f.mem) | ||
| } | ||
| // is_update is not relevant for prepared substrait plans | ||
|
|
| // If the server returned the Dataset Schema or Parameter Binding schemas | ||
| // at creation, they will also be accessible from this object. Close | ||
| // or isUpdate at creation, they will also be accessible from this object. Close | ||
| // should be called when no longer needed. |
There was a problem hiding this comment.
Thanks for the PR and the thorough tests — the feature looks good.
(Edit: withdrawing my earlier concern about the regenerated FlightSql.pb.go toolchain — that's fine as-is, no need to regenerate.)
One thing to tidy before merge — see the inline comment on server_test.go: the isUpdate := … declarations inside the cases shadow the outer var isUpdate *bool, so the trailing if isUpdate != nil block is dead code. Dropping the outer var and that block keeps the helper correct and clear.
I also left an optional, non-blocking note on the IsUpdate() *bool getter shape — take it or leave it.
Happy to approve once the test helper is cleaned up.
| default: | ||
| err = fmt.Errorf("unknown query: %s", query) | ||
| } | ||
| if isUpdate != nil { |
There was a problem hiding this comment.
The isUpdate := … declarations inside the two cases above shadow the outer var isUpdate *bool, so this if isUpdate != nil always sees nil — it's dead code. The cases already set result.IsUpdate directly, so the outer var isUpdate *bool and this block can both be removed.
There was a problem hiding this comment.
Thank you for this review.
Addressed in 3466969
| // IsUpdate returns the server's hint about whether this prepared statement | ||
| // should be executed as an update (true) or query (false). If nil, the server | ||
| // did not provide a hint and the client can choose how to execute the statement. | ||
| func (p *PreparedStatement) IsUpdate() *bool { return p.isUpdate } |
There was a problem hiding this comment.
Returning the internal *bool leaks mutable state (a caller can do *ps.IsUpdate() = false and mutate the statement) and isn't idiomatic for an optional. Prefer IsUpdate() (val bool, ok bool), or return a copy.
There was a problem hiding this comment.
Thank you for pointing this out. I took the idiomatic approach with the double boolean return, see 2c8c152
|
@ennuite are you still working on this? |
|
@zeroshade Sorry, I will resume work on this now and will address the comments this week. |
|
@zeroshade I addressed your review comments, and refactored some other poor patterns in the tests. I will put this as ready for review again once I update apache/arrow-adbc#4161 with the changes I did now and ran end-to-end tests to ensure that everything is ok. |
a26b6b3 to
f6a8588
Compare
zeroshade
left a comment
There was a problem hiding this comment.
Thanks for pushing this through — and thanks especially for the thorough test additions; the prepared-statement integration tests are a genuine gap this fills independently of the feature itself.
I verified the change fairly aggressively and the feature itself is correct. Summary of what I checked:
| Check | Result |
|---|---|
Spec conformance vs upstream apache/arrow format/FlightSql.proto |
optional bool is_update = 4; doc comments match the spec verbatim |
| Upstream status | apache/arrow#49498, merged 2026-06-09 — canonical, not a proposal |
| Generated-file semantic delta (all wire tags / exported methods / types / enum constants, vs true merge-base) | Exactly +1 wire tag, +1 getter, 0 type changes, 0 enum changes |
| Runtime descriptor | #4 is_update kind=bool hasPresence=true |
| Presence round-trip | true, false, and unset all round-trip correctly — importantly false survives the wire, confirming real presence semantics |
go build / go vet / go test ./arrow/flight/... |
All clean; the 4 new tests confirmed executing |
So I want to be clear that the -868 net lines in FlightSql.pb.go is not hiding anything — I checked that specifically.
I'm marking this request changes for one substantive reason only: the unset/nil case is never asserted anywhere. Every new test asserts ok == true, but ok == false is the backward-compatibility contract with the (currently overwhelming majority of) servers that don't send the hint. That's the one path most likely to regress silently and it's the cheapest possible test to add.
Everything else below is either a release-note item or a nit. Two design choices I want to explicitly endorse so they don't get second-guessed in later review rounds: the (val, ok bool) getter shape, and keeping the hint purely informational. Details inline.
| s.Equal(&emptyFlightInfo, info) | ||
| } | ||
|
|
||
| func (s *FlightSqlClientSuite) TestPreparedStatementExecuteWithIsUpdateFalse() { |
There was a problem hiding this comment.
Please add a test for the unset/nil case. This is my only blocking request.
Both new client tests here, and both new server tests, assert ok == true. Nothing anywhere asserts ok == false. That's the path taken by every server that doesn't implement the new field — i.e. the backward-compatibility contract — and it's currently the only untested branch of the getter.
TestPreparedStatementExecute above (line 368) already builds an ActionCreatePreparedStatementResult with no IsUpdate, so this can be as small as adding two lines there:
val, ok := prepared.IsUpdate()
s.False(ok)
s.False(val)Worth locking down explicitly because a future refactor that switched isUpdate *bool to a plain bool would silently turn "no hint" into "is a query" — which is a real behavior change for clients that branch on this — and no existing test would catch it.
There was a problem hiding this comment.
Do we want the s.False(val) check? If s.False(ok), then val is meaningless.
There was a problem hiding this comment.
I addressed this in eb16088
I do not think it makes sense to have the check in val: it being False seems like an implementation detail, and not a part of the contract of the IsUpdate() method. Let me know if you disagree: I can change it to look exactly like your suggestion.
| handle: result.PreparedStatementHandle, | ||
| datasetSchema: dsSchema, | ||
| paramSchema: paramSchema, | ||
| isUpdate: result.IsUpdate, |
There was a problem hiding this comment.
Two small things on this propagation site.
-
Untested.
TestPreparedStatementLoadFromResult(client_test.go:971) is the only coverage forLoadPreparedStatementFromResult, and it doesn't assertIsUpdate. Since that test already constructs the result and checks the other two propagated fields (ParameterSchema,DatasetSchema), adding anIsUpdateassertion there would be consistent and nearly free. (Couldn't leave this as an inline comment on that line — it's outside the diff.) -
Nit, no change required: this is the one place that stores a
*boolowned by the caller, so a caller mutatingresult.IsUpdateafter this call would change the statement's view.Prepare/PrepareSubstraitdon't have this property since they point at their own unmarshaled local. Not worth a defensive copy unless you'd want it forhandletoo — just noting it so the asymmetry is a conscious choice.
There was a problem hiding this comment.
- I addressed this with 2 tests. In
TestPreparedStatementLoadFromResultI let things as they are, and tested with the new field unset, see 8ba207b. Note that this test has the same nuance aboutvalthat I pointed out in feat(arrow/flight/sql): Add is_update field to ActionCreatePreparedStatementResult #732 (comment)
I added a new test that checks that the field is properly loaded also when set, see a334b86
- For that one I purposefully followed the example set by
handle, but I will admit I had my own reservations while doing it.
I will not change this in this PR unless you ask me to, but in your opinion is it worth a follow-up PR to have a defensive copy for both handle and isUpdate? If so I can tackle that after this one
| ParameterSchema *arrow.Schema | ||
| // IsUpdate indicates whether the prepared statement should be executed | ||
| // as an update (true) or query (false). If nil, the client can choose. | ||
| IsUpdate *bool |
There was a problem hiding this comment.
Not a defect, but this needs a release note, because the public API impact is wider than this struct.
types.go:882 has:
type CreatePreparedStatementResult = pb.ActionCreatePreparedStatementResultBeing a type alias, the regenerated proto field lands on a public flightsql symbol too. So this PR adds an exported field to two publicly-reachable structs. Keyed composite literals are unaffected (and that's what I found in the downstream usage I spot-checked), but any downstream unkeyed literal like:
return flightsql.ActionCreatePreparedStatementResult{handle, ds, ps}, nilwill stop compiling. That's permitted under the Go 1 compatibility rules — unkeyed literals of imported struct types are explicitly not covered — and go vet's composites check flags them, so real-world exposure should be low. I don't think it should block the PR, but it's worth a line in the changelog rather than letting someone discover it at upgrade time.
The unconditional result.IsUpdate = output.IsUpdate on both the statement and Substrait branches is correct, by the way — both sides are *bool, so nil flows through and no presence check is needed. Good that the Substrait path wasn't forgotten.
There was a problem hiding this comment.
The comment mentions 2 distinct public structs. For clarity, you are asking about release notes for both of them, right?
I updated the PR description with the breaking change. I looked into how release notes are generated, and it seems like at release time Github will use the PR title in the release notes, but not the description. Given that the description is the place where the breaking change is highlighted, do I need to add release notes elsewhere?
Explanation of the 2 structs
If you agree with my PR description you don't need to read this detail, I leave it for clarity in case I misunderstood your ask.
The type alias refers to flightsql.CreatePreparedStatementResult. This is the raw proto version, what goes in the wire, and I think it's there so that users of the flightsql package don't need to import the proto package directly. In this codebase, this is only used in LoadPreparedStatementFromResult()
But the line about the unkeyed literal is referring to a different struct flightsql.ActionCreatePreparedStatementResult that is unrelated to the alias. It is defined in server.go, and it is a developer-friendly type with the *arrow.Schema objects decoded from wire format. It is used in quite a few places in the codebase.
Both of them fall under the situation of a public struct where unkeyed literals will break, so I put both in the PR description.
Thank you for pointing out the compatibility rules, for reference I found them at https://go.dev/doc/go1compat:
For the addition of features in later point releases, it may be necessary to add fields to exported structs in the API. Code that uses unkeyed struct literals (such as pkg.T{3, "x"}) to create values of these types would fail to compile after such a change. However, code that uses keyed literals (pkg.T{A: 3, B: "x"}) will continue to compile after such a change. We will update such data structures in a way that allows keyed struct literals to remain compatible, although unkeyed literals may fail to compile.
|
Thanks for the very thorough review! I'm going over the comments one by one, I might be a bit slow as I'm ramping up on some of the Go concepts, and your feedback has been a great learning experience. I appreciate your patience as I improve the PR. I will continue working on it in the next few days, and at the end I'll try to tackle #1029 as well. |
|
sounds good! Just ping me here when you're ready for another review! feel free to ask any questions if you run into trouble |
|
Still not ready for a review as I still need to look into the protobuf stuff and the release notes, but a question popped in my mind (which is probably nitpicky). What is your opinion on For the server tests, I'm always using For the client tests:
|
Generally my way of looking at it is pretty much what you describe, whether I find the subsequent tests still interesting/useful if the condition I'm asserting fails. Essentially if I'm testing a condition that if it fails I know that every subsequent assertion in the test will fail, then I just use |
|
@ennuite Converting to draft — this PR doesn't yet meet our Pull Request quality criteria.
See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush. Note: This comment was drafted by an AI-assisted triage tool run by a maintainer, and may contain mistakes. Once you have addressed the points above, an Apache Arrow Go maintainer — a real person — will take the next look at your PR. If anything here looks wrong, say so on the PR and a maintainer will follow up. See CONTRIBUTING.md for the project's contribution conventions. |
3009533 to
c1bc900
Compare
The rebase brought in 9e6dadb which now clears the FlightDescriptor after the schema message, so the record batch send has a nil descriptor. Update TestPreparedStatementExecuteUpdateWithIsUpdateTrue to expect two separate Send calls matching that new behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Addressed all comments and rebased @zeroshade. Let me know what you think! |
zeroshade
left a comment
There was a problem hiding this comment.
LGTM, thanks for all the work here and getting this in!
Rationale for this change
Following the discussion in the mailing list
, this PR introduces a new field in
ActionCreatePreparedStatementResult,optional boolean is_update, that allows servers to indicate to clients the correct execution path for prepared statements.What changes are included in this PR?
The new field
is_updateis added toPreparedStatement, as well as serialization/deserialization and getters.FlightSql.pb.gowas regenerated in order to include the changes in apache/arrow@2cd5ad2, withprotoc-gen-go v1.36.11andprotoc v7.34.0.Are these changes tested?
Yes. Unit tests for the new field were added to
client_test.go, that complement the already existing unit tests for prepared statements queries (there were no tests for prepared statement updates). Integration tests for the new field were added toserver_test.go. These are also the first integration tests for prepared statements.Are there any user-facing changes?
Yes.
There are breaking changes to unkeyed literals for the exported structs
flightsql.ActionCreatePreparedStatementResultandflightsql.CreatePreparedStatementResult.There are also user-facing but non breaking changes: an extension to prepared statements was added to allow for reading/writing to the new field.
AI Disclaimer
This change was created with AI assistance (Augment Code and Claude Code and Codex). All lines were manually reviewed by a human. The output is not copyrightable subject matter.
Closes #705